Skip to content

Rust memory engine: 6-layer parallel recall with full ORM integration#260

Merged
joelteply merged 4 commits intomainfrom
feature/cognition-optimization
Jan 30, 2026
Merged

Rust memory engine: 6-layer parallel recall with full ORM integration#260
joelteply merged 4 commits intomainfrom
feature/cognition-optimization

Conversation

@joelteply
Copy link
Contributor

Summary

  • Built complete Rust memory system: MemoryCorpus, 6-layer multi-recall (Core, Semantic, Temporal, Associative, DecayResurface, CrossContext), consciousness context with temporal + cross-context awareness
  • Wired Rust ↔ TypeScript: bulk corpus loading from ORM at startup, incremental COW append on every DB write (Hippocampus memories + timeline events)
  • DashMap<Arc> concurrency model: lock-free reads, copy-on-write writes, snapshot isolation
  • 293 Rust tests passing, TypeScript compiles clean

Test plan

  • cargo test --release -p continuum-core — 277 lib tests pass
  • cargo test --release -p continuum-core --test memory_recall_accuracy — 16 recall accuracy tests pass
  • npm run build:ts — TypeScript compilation succeeds
  • Deploy with npm start and verify persona logs show corpus loading
  • Send chat message and verify RAG sources return memories

Joel added 4 commits January 30, 2026 02:22
Rust engine (4 new files, 1839 lines):
- channel_types.rs: ActivityDomain enum, QueueItemBehavior trait with RTOS aging
- channel_items.rs: Voice (always urgent, never kicked), Chat (per-room consolidation),
  Task (dependency-aware, overdue urgency) + ChannelEnqueueRequest discriminated union
- channel_queue.rs: Generic priority queue with capacity kick management
- channel_registry.rs: Domain routing, service_cycle() with state-gated scheduling

IPC layer (5 new commands):
- channel/enqueue, channel/dequeue, channel/status, channel/service-cycle, channel/clear
- Per-persona registry auto-creation with Arc<Mutex<HashMap>>

TypeScript bridge:
- RustCoreIPC.ts: 5 low-level IPC methods
- RustCognitionBridge.ts: 4 high-level bridge methods with forensic logging
- PersonaInbox.ts: Rust-first routing with TS fallback
- QueueItemTypes.ts: toChannelEnqueueRequest() factory
- 6 ts-rs generated type exports

232 tests pass, zero clippy warnings in new code.
CNS service cycle now asks Rust "what should I process next?" via IPC:
- serviceViaRust(): calls rustBridge.serviceCycle() for <1ms scheduling
- Rust handles consolidation, state gating, urgent-first ordering
- TS just executes what Rust decided (LLM calls, DB, tools)
- Falls back to TS channels if Rust bridge unavailable

Wiring:
- CNSTypes.ts: added optional rustBridge to CNSConfig
- CNSFactory.ts: passes rustCognitionBridge from PersonaUser
- PersonaUser.ts: added nullable rustCognitionBridge getter,
  wired inbox.setRustBridge() after bridge initialization
- QueueItemTypes.ts: added fromRustServiceItem() converter
  (Rust camelCase JSON → TS QueueItem discriminated union)

232 Rust tests pass, TypeScript builds clean.
… forwarding

- Delete 9 dead TS files: channel queue system (6 files) and cognitive
  schedulers (3 files) now fully replaced by Rust cognition engine
- Slim down CNS, PersonaInbox, PersonaAutonomousLoop, PersonaUser to
  remove all fallback/legacy TS scheduling paths
- Fix @reactive() and @attr() decorators: TC39 class fields shadow
  Lit's prototype accessor from createProperty(), preventing re-renders.
  queueMicrotask removes the own property after init so Lit's setter
  triggers requestUpdate() automatically.
- Forward all transcription captions to all participants (including
  speaker's own) — captions confirm system is working, audio echo is
  handled by mix-minus in the Rust mixer, not transcription filtering.
Phase A: Built complete memory system in Rust (corpus, recall, consciousness)
- MemoryCorpus: per-persona in-memory data (memories + embeddings + timeline events)
- 6-layer multi-recall: Core, Semantic, Temporal, Associative, DecayResurface, CrossContext
- Consciousness context: temporal awareness + cross-context peripheral awareness
- DashMap<Arc<MemoryCorpus>> concurrency: lock-free reads, COW writes
- Deterministic embeddings for offline recall (no external model dependency)
- 293 tests passing (277 lib + 16 recall accuracy integration tests)

Phase B: Wired Rust ↔ TypeScript via IPC
- Cleaned dead TS methods (memoryInit/Store/Recall/RecordEvent/Consolidate)
- Added memoryLoadCorpus: bulk ORM → Rust at startup
- Added memoryAppendMemory/Event: incremental COW updates on DB writes
- Hippocampus: appends to Rust corpus after every DataCreate to longterm.db
- UnifiedConsciousness: appends timeline events to Rust corpus after recording
- PersonaUser: loads full corpus from ORM at startup, wires bridge to consciousness
- SemanticMemorySource + GlobalAwarenessSource: call Rust recall (sub-ms response)
Copilot AI review requested due to automatic review settings January 30, 2026 23:11
@joelteply joelteply merged commit 1341420 into main Jan 30, 2026
2 of 5 checks passed
@joelteply joelteply deleted the feature/cognition-optimization branch January 30, 2026 23:11
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a complete Rust-based memory system with 6-layer parallel recall and multi-channel queue management. The implementation migrates critical memory operations from TypeScript to Rust for performance improvements (3-60s → ~30ms for memory operations).

Changes:

  • Built complete Rust memory subsystem: MemoryCorpus, 6-layer recall (Core, Semantic, Temporal, Associative, DecayResurface, CrossContext), consciousness context builder, timeline queries
  • Integrated Rust ↔ TypeScript: bulk corpus loading from ORM at startup, incremental COW append on DB writes, DashMap<Arc> concurrency model
  • Replaced TS-based channel system with Rust multi-domain queue: AUDIO/CHAT/BACKGROUND channels with item polymorphism, RTOS aging, consolidation

Reviewed changes

Copilot reviewed 67 out of 76 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
continuum-core/src/memory/*.rs New Rust memory engine with types, corpus, embedding, consciousness, timeline, and cache modules
continuum-core/src/persona/channel_*.rs New Rust channel system replacing TS BaseQueueItem/ChannelQueue/ChannelRegistry
continuum-core/src/main.rs Initialize FastEmbed embedding provider and PersonaMemoryManager
continuum-core/tests/common/mod.rs Shared IPC client utilities with length-prefixed binary framing protocol
bindings/*.ts Generated TypeScript types for Rust memory and channel systems
RustCognitionBridge.ts Added memory and channel system IPC methods
PersonaUser.ts Integrated corpus loading from ORM and Rust bridge wiring
PersonaCentralNervousSystem.ts Replaced TS scheduling with Rust-delegated service_cycle
CNSFactory.ts Simplified to create Rust-delegated CNS instances
QueueItemTypes.ts Added conversions between TS queue items and Rust channel enqueue requests
PersonaInbox.ts Routes items through Rust channel system via IPC
SemanticMemorySource.ts Delegates to Rust multi-layer recall via IPC
GlobalAwarenessSource.ts Delegates to Rust consciousness context builder via IPC
Deleted files Removed TS channel system (BaseQueueItem.ts, ChannelQueue.ts, etc.) and cognitive schedulers (ICognitiveScheduler.ts, HeuristicCognitiveScheduler.ts, etc.)
Files not reviewed (1)
  • src/debug/jtag/package-lock.json: Language not supported
Comments suppressed due to low confidence (17)

src/debug/jtag/system/user/server/PersonaUser.ts:23

  • Unused import Commands.
import { Commands } from '../../core/shared/Commands';

src/debug/jtag/system/user/server/PersonaUser.ts:38

  • Unused import TaskEntity.
import { TaskEntity } from '../../data/entities/TaskEntity';

src/debug/jtag/system/user/server/PersonaUser.ts:39

  • Unused import taskEntityToInboxTask.
import { taskEntityToInboxTask } from './modules/QueueItemTypes';

src/debug/jtag/system/user/server/PersonaUser.ts:42

  • Unused import ChatRAGBuilder.
import { ChatRAGBuilder } from '../../rag/builders/ChatRAGBuilder';

src/debug/jtag/system/user/server/PersonaUser.ts:47

  • Unused import AIDecisionService.
import { AIDecisionService, type AIDecisionContext } from '../../ai/server/AIDecisionService';

src/debug/jtag/system/user/server/PersonaUser.ts:49

  • Unused import CoordinationDecisionLogger.
import { CoordinationDecisionLogger, type LogDecisionParams } from '../../coordination/server/CoordinationDecisionLogger';

src/debug/jtag/system/user/server/PersonaUser.ts:61

  • Unused import AI_DECISION_EVENTS.
import {
  AI_DECISION_EVENTS,
  type AIEvaluatingEventData,
  type AIDecidedRespondEventData,
  type AIDecidedSilentEventData,
  type AIGeneratingEventData,
  type AICheckingRedundancyEventData,
  type AIPostedEventData,
  type AIErrorEventData
} from '../../events/shared/AIDecisionEvents';

src/debug/jtag/system/user/server/PersonaUser.ts:68

  • Unused import AI_LEARNING_EVENTS.
import {
  AI_LEARNING_EVENTS,
  type AITrainingStartedEventData,
  type AITrainingCompleteEventData,
  type AITrainingErrorEventData,
  type AIInteractionCapturedEventData
} from '../../events/shared/AILearningEvents';

src/debug/jtag/system/user/server/PersonaUser.ts:82

  • Unused import PersonaGenome.
import { PersonaGenome, type PersonaGenomeConfig } from './modules/PersonaGenome';

src/debug/jtag/system/user/server/PersonaUser.ts:95

  • Unused import CognitionLogger.
import { CognitionLogger } from './modules/cognition/CognitionLogger';

src/debug/jtag/system/user/server/modules/PersonaMessageEvaluator.ts:31

  • Unused import AIDecisionService.
import { AIDecisionService } from '../../../ai/server/AIDecisionService';

src/debug/jtag/system/user/server/modules/PersonaMessageEvaluator.ts:53

  • Unused import getStageStatus.
import {
  COGNITION_EVENTS,
  type StageCompleteEvent,
  calculateSpeedScore,
  getStageStatus
} from '../../../conversation/shared/CognitionEventTypes';

src/debug/jtag/system/user/server/modules/cognitive/memory/Hippocampus.ts:18

  • Unused import DATA_COMMANDS.
    src/debug/jtag/system/user/server/modules/cognitive/memory/Hippocampus.ts:20
  • Unused import Commands.
    src/debug/jtag/system/user/server/modules/cognitive/memory/Hippocampus.ts:27
  • Unused import generateUUID.
    src/debug/jtag/system/user/server/modules/cognitive/memory/Hippocampus.ts:39
  • Unused import RawMemoryAdapter.
    src/debug/jtag/system/user/server/modules/cognitive/memory/Hippocampus.ts:41
  • Unused import DataDaemon.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@@ -69,7 +69,7 @@ import {
import { Events } from '../../core/shared/Events';
import { EVENT_SCOPES } from '../../events/shared/EventSystemConstants';
Copy link

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused import EVENT_SCOPES.

Suggested change
import { EVENT_SCOPES } from '../../events/shared/EventSystemConstants';

Copilot uses AI. Check for mistakes.
import type { DataListParams, DataListResult } from '../../../commands/data/list/shared/DataListTypes';
import { DataList, type DataListParams, type DataListResult } from '../../../commands/data/list/shared/DataListTypes';
import type { StageCompleteEvent } from '../../conversation/shared/CognitionEventTypes';
import { calculateSpeedScore, getStageStatus, COGNITION_EVENTS } from '../../conversation/shared/CognitionEventTypes';
Copy link

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused imports COGNITION_EVENTS, calculateSpeedScore, getStageStatus.

Suggested change
import { calculateSpeedScore, getStageStatus, COGNITION_EVENTS } from '../../conversation/shared/CognitionEventTypes';

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant